SYSTEMS AND METHODS FOR CONSTRUCTING AN 
IMAGE HAVING AN EXTENDED DEPTH OF FIELD 



BACKGROUND OF THE INVENTION 

1. Field of Invention 

[0001] This invention relates to image-processing systems and methods 
usable to construct a composite image. 

2. Description of Related Art 

[0002] High resolution machine vision systems, and especially microscopic 
machine vision systems, often provide a high-resolution, well-focused image of a 
small portion of a three-dimensional object. However, this is accomplished at the 
expense of the depth of field, also called the depth of focus, of the surrounding overall 
image. However, in some machine vision system applications, it is useful both to 
make high-resolution well-focused microscopic examination and/or record of a small 
portion of an object and to clearly observe or record the surrounding overall image. 

[0003] Methods are known for combining, or fusing, separately focused 
images of a three-dimensional scene or object to accomplish this useful goal. For 
example, U.S. Patent 4,141,032 to Haeusler discloses focusing through an object at a 
plurality of levels to produce a plurality of images, filtering each image with a high 
pass filter, and summing the images passed by the filtering step to produce a 
composite image containing only the sharp details. Similarly, U.S. Patent 4,584,704 
to Ferren discloses obtaining visual in-focus "slices" of an entire image field. When 
processing the slices as video scan line signals, the signals pass through a high pass 
filter. This provides signals which may be discriminated to remove all but the spikes, 
which identify in-focus edges of objects. Signal information between two successive 
spikes (edges) may also be regarded as part of an in-focus object. The high pass filter 
may be combined with other filters. The filters may be made adaptive. 

[0004] In general, such methods that are based on such relatively simple 
high-pass filtering are relatively fast. However, the composite images obtained are 
sensitive to the selected filter parameters and/or signal thresholds employed to 
discriminate between in-focus and out-of-focus features. As such, these methods are 
not robust for an unpredictable variety of source image objects and/or surface 
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characteristics, and may introduce undesirable information loss and/or admit 
undesirable high spatial frequency artifacts from out-of-focus image portions. 

[0005] Methods based on various multi-resolution spatial filtering 
techniques are also known. For example, U.S. Patent 4,661,986 to Adelson lists 
5 numerous papers by Burt describing various aspects of the Burt pyramid method, and 
discloses a variation of that method. Adelson discloses dividing respective spatial- 
frequency spectrums of M images into M substantially similar assemblages of N 
separate specified pixel sample sets that define N spatial frequency bands. A single 
one of the corresponding samples is selected to derive respective single sets of 

10 improved focus pixel samples for each of the N bands. Corresponding pixel samples 
of the respective single sets are then combined to derive the improved-focus two- 
dimensional image. 

[0006] U.S. Patent 5,325,449 to Burt discloses an image fusion method 
general enough to fuse not only differently-focused images, but qualitatively different 

15 images provided by qualitatively different types of sensing/imaging systems. Burt 

notes that image fusion is successful to the extent that the composite image retains all 
useful information, does not contain artifacts, and looks natural. Burt notes that 
previous pyramid techniques, including multi-resolution spatial frequency techniques, 
have produced noticeable artifacts in composite images. Burt discloses an improved 

20 pattern selective method based upon using oriented functions, such as a disclosed 

gradient function, which improves the retention of edge-like source image patterns in 
the composite image. This method is enhanced by a local saliency analysis to refine 
the composite image. 

[0007] In general, the pyramid methods of Adelson and Burt can be 

25 relatively fast, and relatively robust. However, these pyramid methods are primarily 
based on a single type of mathematical operation which is applied uniformly 
throughout all images. Thus, the methods remain sensitive, for example, to 
undesirable high-frequency artifacts which may be produced by unfocused edges in 
otherwise featureless and/or out-of-focus portions of the source image. 

30 [0008] Convolution methods and wavelet-based methods are also known to 

be usable to derive composite images. Another known "image fusion" method relies 
on formulating a multi-component energy function and determining the image fusion 
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result which minimizes the energy function. However, all of these methods generally 
apply a single type of mathematical operation uniformly throughout all images. 
Depending on the implementation details, such methods are either computationally- 
intensive, relatively slow, and/or they also share the drawbacks of the previously- 
5 described methods. 

[0009] Further, numerous methods are known that both identify edges in 
single images and segment single images. However, such methods have failed to 
consider how to select among multiple source images to create a composite image in a 
way that anticipates and suppresses the artifacts that are likely to exist in out-of-focus 
1 0 portions of an image. 

SUMMARY OF THE INVENTION 
[0010] Thus, considering that a composite image is successful to the extent 
that the composite image retains all useful information, does not contain artifacts, and 
looks natural, and furthermore considering the commercial value and convenience of 

1 5 rapid computation, and furthermore considering the need for robust and reliable 

results on variety of unpredictable workpiece features and configurations, there is a 
need for an improved, commercially-appealing solution that balances efficacy against 
throughput time and provides a high-resolution well-focused image of a small portion 
of a three-dimensional object while also allowing the user to clearly observe or record 

20 the area surrounding that small portion of the three-dimensional object. This problem 
is particularly significant in automated machine vision systems used for microscopic 
inspection and measurement. In such systems, there is a need for a composite image 
construction method that is relatively fast, preserves detail, provides robust results on 
variety of unpredictable workpiece features and configurations, and that tends to 

25 suppress or reduce out-of-focus artifacts. 

[0011] This invention provides systems and methods that construct a 
desirable composite image having an extended depth of focus in a relatively short 
time, and preserves the image detail present in a plurality of source images. 

[0012] This invention separately provides systems and methods that 

30 construct a desirable composite image reliably and robustly, for a variety of 
unpredictable workpiece features and configurations. 
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[0013] This invention further provides systems and methods that are readily 
adaptable to construct a desirable composite image even more reliably, robustly, and 
in an even shorter time, when certain workpiece features are predictable. 

[0014] This invention separately provides systems and methods that 
construct a desirable composite image while suppressing likely artifacts arising from 
out-of-focus features. 

[0015] In various exemplary embodiments of the systems and methods 
according to this invention, an object is placed within the overall field of view of a 
vision system. A plurality of source images of the object are captured, where each 
captured source image exhibits a different plane of the object in the nominal focus 
plane of the source image. The source images are either acquired with spatial 
congruence, or their data representations are mutually adjusted to be spatially 
congruent. A composite image having an extended depth of focus is constructed by 
identifying well-focused edges and/or boundaries in the source images. Each 
particular edge or boundary in the composite image is determined based generally on 
the source image with the best-focused instance of each particular edge or boundary. 
The composite image is also constructed by identifying well-focused surfaces in the 
source images. Each particular surface portion in the composite image is determined 
based generally on the source image with the best-focused instance of each particular 
surface portion. 

[0016] In various exemplary embodiments, the composite image is 
generated hierarchically, in that the determined edges or boundaries are given priority 
in the composite image regardless of whether an apparently well-focused surface 
portion is identified at the same spatial location as an edge or boundary. This tends to 
suppress artifacts in the composite image in the vicinity of boundaries and edges. 

[0017] In various other exemplary embodiments of the systems and 
methods according to this invention, additional buffer zones are established 
immediately adjacent to determined edges or boundaries. The composite image 
generation in these buffer zones suppresses artifacts which might otherwise be created 
by out-of-focus edges "bleeding over into surface regions" in some source images and 
producing anomalous results during the surface analysis. 
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[0018] In various other exemplary embodiments of the systems and 
methods according to this invention, the some of the determined well-focused surface 
portions which initially provide a basis for the composite image are modified, based 
on consideration of the plane of their corresponding source image, relative to the 
5 source image plane(s) of neighboring surface portions. 

[0019] In contrast to the previously discussed conventional methods, in the 
various exemplary embodiments of the systems and methods according to this 
invention, the composite image can be constructed relatively quickly while preserving 
detail in the composite image. The resulting composite images can be constructed 
1 0 robustly in view of a variety of unpredictable workpiece features and configurations, 
while out-of- focus artifacts in the resulting composite images tend to be suppressed or 
reduced. 

[0020] These and other features and advantages of this invention are 
described in, or are apparent from, the following detailed description of various 
15 exemplary embodiments of the systems and methods according to this invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0021] Various exemplary embodiments of this invention will be described 
in detail, with reference to the following figures, wherein: 

Fig. 1 is a block diagram of one exemplary embodiment of a vision system 
20 having a composite image construction system according to this invention; 

Fig. 2 is a block diagram outlining in greater detail exemplary embodiments of 
the composite image processor of Fig. 1 according to this invention; 

Fig. 3 is a flowchart outlining one exemplary embodiment of a method for 
generating a composite image according to this invention; 
25 Fig. 4 is a flowchart outlining in greater detail one exemplary embodiment of 

the step of analyzing the source images using a technique sensitive to and indicative of 
potential boundary pixels and/or edge pixels of Fig. 3 according to this invention; 

Fig. 5 is a flowchart outlining in greater detail one exemplary embodiment of 
the step of determining or selecting a set of edge pixels of Fig. 3 according to this 
30 invention; 

Fig. 6 is a flowchart outlining in greater detail one exemplary embodiment of 
the step of refining the composite edge map of Fig. 5 according to this invention; 
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Fig. 7 is a flowchart outlining in greater detail one exemplary embodiment of 
the step of analyzing the source images using a technique sensitive to and indicative of 
potential surface pixels of Fig. 3 according to this invention; 

Fig. 8 is a flowchart outlining in greater detail one exemplary embodiment of 
5 the step of determining or selecting a set of surface pixels of Fig. 3 according to this 
invention; 

Fig. 9 is a flowchart outlining in greater detail one exemplary embodiment of 
the step of refining the composite surface map of Fig. 8 according to this invention; 

Fig. 10 is a first source image having a focal plane located at a first plane of an 

10 object; 

Fig. 1 1 is a second source image having a focal plane located at a second plane 
of an object; 

Fig. 12 is a composite image map based on a set of source images including 
the source images of Figs. 9 and 10, determined according to this invention; and 

15 Fig. 13 is a composite image, based on the composite image map of Fig. 12 

and a set of source images including the source images of Figs. 9 and 10, determined 
according to this invention. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
[0022] For simplicity and clarification, the operating principles, and design 

20 factors of various exemplary embodiments of the systems and methods according to 
this invention are explained with reference to one exemplary embodiment of a vision 
system 100, as shown in Fig. 1 . The basic explanation of the operation of the vision 
system shown in Fig. 1 is applicable for the understanding and design of any vision 
system that incorporates the systems and/or methods of this invention. Although the 

25 systems and methods of this invention are described in conjunction with this specific 
vision system 100, the systems and methods according to this invention can be used 
with any other known or later-developed vision system. 

[0023] Fig. 1 shows a vision system 100. As shown in Fig. 1, the vision 
system 100 includes a vision system components portion 110 and a control system 

30 portion 120. The control system portion 120 includes elements, including a composite 
image processor 150, which constitute a composite image construction system usable 
in accordance with this invention. The control system portion 120 further includes 
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elements, including a part program executor and/or generator 170 described below, 
which constitute a control instruction generation system usable in accordance with this 
invention. 

[0024] The vision system components portion 110 includes a stage 111. 
5 The stage 1 1 1 may be movable relative to the remainder of the vision system 

components portion 1 1 0, in a plane normal to a z-axis 300, for example along an x- 
axis 310. A part 20 to be imaged using the vision system 100 is placed on the stage 
111. The part 20 shown in Fig. 1 includes a number of relatively planar surface 
regions 24, 26 and 28, a number of edges 25 and 27 and at least one inclined surface 

1 0 region 21. A light source 117 can be used to illuminate the part 20. The light source 
117 may be a simple surface light, a ring light or a programmable ring light, and may 
include other lights sources, such as a stage light and/or a coaxial light . 

[0025] The light from the light source 117, after illuminating the part 20, 
passes through a lens system 1 13, and is gathered by a camera system 1 14 to generate 

15 an image of the part 20. The lens system 113 and the camera system 1 1 4 are fastened 
to a housing/frame 1 15 to form an imaging assembly 1 12. The imaging assembly 1 12 
is movable along the z-axis 300 by an actuator 116. By moving the imaging assembly 
112 along the z-axis 300, the focal plane of the lens system 1 1 3 is moved along the z- 
axis 300 relative to the part 20. Thus, it is possible to move the imaging assembly 112 

20 over a chosen range and capture a set of source images of the part 20, where different 
portions of the part 20 are in focus in different ones of the set of source images. 
Furthermore, the set of source images which are captured in this manner will, in 
general, be substantially spatially congruent. 

[0026] Each image acquired by the camera system 1 14 is output on a signal 

25 line 31 to the control system portion 120. As shown in Fig. 1, the control system 
portion 120 includes a controller 125, an input/output interface 130, a memory 140, 
the part program executor and/or generator 180, a CAD file feature extractor 190, and 
the composite image processor 150, each interconnected either by a data/control bus 
136 or by direct connections between the various elements. The signal line 31 to and 

30 from the camera system 114, the signal line 32 to and from the actuator 116, and the 
signal line 33 to and from the light source 117 are connected to the input/output 
interface 130. In various exemplary embodiments, the input/output interface 130 may 
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include one or more of an imaging control interface portion (not shown) usable to 
interact with the camera system 1 14, a motion control interface portion (not shown) 
usable to interact with the actuator 116, and a lighting control interface portion (not 
shown) usable to interact with the light source 117. 
5 [0027] A display device 134 and one or more input devices 135 also can be 

connected to the input/output interface 130over a signal line 34 and one or more 
signal lines 35, respectively. The display 134 and the one or more input devices 135 
can be used to view, create and modify part programs, to input CAD file information 
regarding the part 20, to view the source images captured by the camera system 114, 
10 to view composite images, and/or to directly operate the control system portion 120 
and/or the vision system components 110. However, it should be appreciated that, in 
a fully automated system having a predefined part program, the display 134 and/or the 
one or more input devices 135, and the corresponding signal lines 34 and/or 35 may 
be omitted. 

1 5 [0028] The memory 140 records and holds data as necessary for the 

operation of the vision system 100. In various exemplary embodiments, the memory 
140 includes one or more of a source image memory portion 141 , a part program 
memory portion 142, a composite image processing memory portion 143, and/or a 
composite image memory portion 144. The source image memory portion 141 stores 

20 source images captured using the camera system 114 when operating the vision 

system 100. The part program memory portion 142 stores one or more part programs 
used to control the operation of the vision system 100 for particular types of parts. 
The composite image processing memory portion 143 contains data recorded and/or 
created as needed during the operations of the composite image processor 150, as 

25 described in greater detail below. The composite image memory portion 144 contains 
composite image data representations and/or composite images which are output by 
the composite image processor 150. 

[0029] The exemplary composite image processor 150 as included in the 
control system portion 120 shown in Fig. 1, is one exemplary embodiment of a 

30 composite image processor usable according to the systems and methods of this 
invention. The exemplary composite image processor 150 of Fig. 1 includes an 
edge/boundary processing circuit or software routine 160, and a surface processing 
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circuit or software routine 170, each interconnected either by a data/control bus 137, 
by one or more application programming interfaces or by direct connections between 
the various circuits or software routines. The data/control bus 137 and the 
data/control bus 136 may be the identical data/control bus in various exemplary 
5 embodiments of the control system portion 120. 

[0030] The exemplary edge/boundary processing circuit or software routine 
160 shown in Fig. 1 processes source image data from a plurality of source images 
and determines a set of edge or boundary pixels in the composite image based on the 
analyzed source images. That is, the edge/boundary processing circuit or software 

10 routine 160 is operable to perform a type of analysis of the set of source images usable 
to determine a first set of pixels of the composite image corresponding to edges and/or 
boundaries in the composite image. The exemplary surface processing circuit or 
software routine 170 shown in Fig. 1 processes source image data from a plurality of 
source images and determines a set of surface pixels in the composite image based on 

15 the analyzed source images. Similarly, the surface processing circuit or software 

routine 170 is operable to perform a type analysis of the set of source images usable to 
determine a second set of pixels of the composite image corresponding to surface 
regions in the composite image. The various elements and operation of the composite 
image processor 150 are described in further detail elsewhere herein. 

20 [0031] It should be appreciated that the foregoing description of the vision 

system 100 usable according to this invention generally describes a system suitable for 
automatic program operation. However, the vision system 1 00 used with this 
invention may also operate substantially the same when the commands, including 
composite image construction commands, are issued manually through the one or 

25 more input devices 135 during manual or stepwise operation of the vision system 100. 
Furthermore, although the vision system 1 00 is capable of adapting to various 
components and operating parameters, the following discussion generally assumes 
that the configuration of the vision system components portion 110 and the workpiece 
under inspection are already known and/or are predictable. In addition, any necessary 

30 operating conditions which are not specifically discussed are assumed to have been 

met according to well-known procedures for the operation of machine vision systems. 
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[0032] When a machine vision system operator wishes to construct a 
composite image of a specific portion of an object, such as a portion of the part 20 
mounted on the stage 1 1 1, the user operates the vision system 100 to position that 
portion of the object within the field of view of the vision system 100. Various image 
5 acquisition parameters, for example, the lighting settings, and the positioning and/or 
focus of the imaging assembly 1 12, are chosen to provide an acceptable set of source 
images. These parameters, positioning and/or focus of the imaging assembly 1 12 are 
obtainable either by operator trial and error or with the assistance of a variety of 
automated "tools" available on a variety of commercially-available machine vision 

10 systems, such as the Quick Vision series of vision inspection machines available from 
Mitutoyo America Corporation (MAC), located in Aurora, IL. 

[0033] By moving the imaging assembly 112 along the z-axis 300, the focal 
plane of the lens system 1 13 is moved along the z-axis 300 relative to the part 20. 
Thus, it is possible to move the imaging assembly 112 over a chosen range and 

15 capture a set of source images of the part 20, where, in general, each important feature 
of the desired portion of the part 20 will be in focus in at least one of the different 
source images. The chosen range may be divided into a plurality of equidistant 
movements along the z-axis in order to acquire the plurality of source images. 
Alternatively, z-axis positions that allow important features of the part 20, such as the 

20 relatively planar surface regions 24, 26 and 28, for example, to be specifically focused 
on or a combination of these approaches, may be chosen. The plurality of source 
images may be stored in the source image memory portion 141 . 

[0034] For the exemplary configuration of the vision system 1 00 shown in 
Fig. 1, the set of source images which are captured in this manner will be substantially 

25 spatially congruent. That is, a particular spatial location in one source image will 

correspond to the same portion of the object at the same spatial location in any of the 
other source images. For purposes of composite image processing, if it is not assured 
that the source images are spatially congruent, the plurality of source images may be 
analyzed by well-known techniques, such as, for example, by image correlation or 

30 other pattern-matching techniques so that the spatial locations of the plurality of 
source images may be adjusted to provide for spatial congruence. The spatial 
congruence/feature congruence processing circuit or software routine 152 of the 
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composite image processor 150 can be used to determine if the images are spatially 
congruent and/or to implement one or more of these techniques for adjusting the 
spatial locations to obtain spatial congruence. 

[0035] If the operator desires to obtain a composite image which 
5 corresponds to only a portion of a full source image then, by using the display device 
134 and/or an input device 135 of the vision system 100, the operator may choose 
and/or define the desired portion of the object 20, for example, by using a so called 
"box tool" on the display device 134. In this case, any subsequent composite image 
processing operations may be restricted to the corresponding spatial locations. 

10 [0036] Any or all of the foregoing operator actions or instructions may be 

recorded by the part program generator and executor 170, and the results stored in the 
part program memory portion 142 for later recall and automatic execution by the part 
program generator and executor 170, for example to inspect and/or make a record of a 
different specimen of the part 20. The foregoing describes one exemplary way of 

1 5 providing a plurality of source images usable according to the systems and methods of 
this invention. However, it should be appreciated that the systems and methods of this 
invention are useable with any similar plurality of source images, regardless of the 
details of the imaging system, the portion of the electromagnetic spectrum used to 
acquire the plurality of source images, and/or the operations used to acquire and/or 

20 prepare the plurality of source images. 

[0037] Fig. 2 is a block diagram outliningexemplary embodiments of the 
edge/boundary processing circuit or software routine 160 and the surface processing 
circuit or software routine 170 of the composite image processor 150 of Fig. 1 
according to this invention. As shown in Fig. 2, the edge/boundary processing circuit 

25 or software routine 160 includes a source image edge/boundary processing circuit or 
software routine 163 and a composite image edge/boundary determining circuit or 
software routine 166. The surface processing circuit or software routine 170 includes 
a source image surface processing circuit or software routine 173 and a composite 
image surface determining circuit or software routine 176. 

30 [0038] It should be appreciated that the particular structures shown in Fig. 2 

for the edge/boundary processing circuit or software routine 160 and the surface 
processing circuit or software routine 170 are independent of each other. Thus, while 
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specific exemplary structures for both of the edge/boundary processing circuit or 
software routine 160 and the surface processing circuit or software routine 170 are 
shown together in Fig. 2, each of these specific structures of the edge/boundary 
processing circuit or software routine 160 and the surface processing circuit or 
5 software routine 1 70 can be implemented separately of the other, and combined with 
other exemplary structures of the other of the edge/boundary processing circuit or 
software routine 160 and the surface processing circuit or software routine 170. Thus, 
the specific combination of structures of the edge/boundary processing circuit or 
software routine 160 and the surface processing circuit or software routine 170 shown 

10 in Fig. 2 should be understood as exemplary only, and not limiting or essential. 

[0039] The source image edge/boundary processing circuit or software 
routine 163 is usable to process and/or characterize the source image data from a 
plurality of source images. The composite image edge/boundary determining circuit 
or software routine 166 determines a set of edge or boundary pixels in the composite 

1 5 image based on the results of the edge/boundary processing circuit or software routine 
163. In various exemplary embodiments, the edge/boundary processing circuit or 
software routine 163 may include an analysis portion (not shown) usable to analyze or 
characterize spatial locations in the source images and provide results indicative of 
potential edge or boundary pixels in the source images. 

20 [0040] In various other exemplary embodiments, the source image 

edge/boundary processing circuit or software routine 163 may further include a 
validating portion (not shown) usable to screen out some of the results indicative of 
potential edge or boundary pixels in the source images. If implemented, the validating 
portion leaves only the results actually indicating acceptable edge or boundary pixels 

25 in any of the source images. Further, in various exemplary embodiments, the 

composite image edge/boundary determining circuit or software routine 166 may 
include a selector portion (not shown) usable to select or qualify the best edge or 
boundary pixels for the composite image from among the results provided by the 
edge/boundary processing circuit or software routine 163. In various other exemplary 

30 embodiments, the composite image edge/boundary determining circuit or software 

routine 166 may further include a refinement portion (not shown) usable to refine the 
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results from the selector portion, for example, to reject image artifacts not otherwise 
rejected by the systems and methods according to this invention. 

[0041] The source image surface processing circuit or software routine 173 
is usable to process and/or characterize source image data from a plurality of source 
5 images. The composite image surface determining circuit or software routine 176 
determines a set of surface pixels in the composite image based on the results of the 
surface processing circuit or software routine 173. In various exemplary 
embodiments, the surface processing circuit or software routine 173 may include an 
analysis portion (not shown) usable to analyze or characterize spatial locations in the 
10 source images and provide results indicative of potential surface pixels in the source 
images. 

[0042] Further, in various exemplary embodiments, the composite image 
surface determining circuit or software routine 176 may include a selector portion (not 
shown) usable to select or qualify the best edge or boundary pixels for the composite 

15 image from among the results provided by the surface processing circuit or software 
routine 173. In various other exemplary embodiments, the composite image surface 
determining circuit or software routine 176 may further include a refinement portion 
(not shown) usable to refine the results from the selector portion, for example, to 
reject image artifacts not otherwise rejected by the systems and methods according to 

20 this invention. 

[0043] In various other exemplary embodiments, the composite image 
processor 150 may further include auxiliary circuits or software routines (not shown) 
including one or more of a spatial congruence and/or feature congruence processing 
circuit or software routine, a composite image refinement processing circuit or 

25 software routine, and a composite image storage and/or output processing circuit or 
software routine which are each interconnected either by a data/control bus, by one or 
more application programming interfaces or by direct connections between the various 
circuits or software routines. The operations performed by these auxiliary circuits or 
software routines are described in greater detail further below. 

30 [0044] Fig. 3 is a flowchart outlining one exemplary embodiment of a 

method for generating a composite image according to this invention. Beginning in 
step SI 00, operation continues to step S200, where vision system operating 
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parameters usable to acquire a plurality of source images of a desired portion of an 
object are defined. The vision system operating parameters can be defined manually, 
in a semi-automated fashion, or fully automatically. The general characteristics of the 
plurality of source images to be obtained through the effects of these parameters have 
5 been previously described. Next, in step S3 00, the plurality of source images are 

acquired and their representative data is stored. Then, in step S400, if it is not assured 
that the acquired source images are aligned, that is, spatially congruent, as outlined 
above, the plurality of source images are analyzed by any known or later-developed 
technique, such as, for example by image correlation or any other pattern-matching 

10 technique. The representative data of the plurality of source images is adjusted based 
on this analysis to place the plurality of images into spatial congruence. Operation 
then continues to step S500. 

[0045] In step S500, if a composite image is desired for a portion of an 
object which corresponds to only a portion of the full source image frame, then the 

15 spatial location range for which a composite image is desired is defined. The bounds 
of this portion of the full source image frame can be defined manually, in a semi- 
automated fashion, or fully automatically. The subsequent composite image 
processing operations may be restricted to correspond to the defined bounds, or spatial 
location range, of the portion to reduce processing time. Then, in step S600, each of 

20 the plurality of source images is characterized and/or analyzed using a technique that 
is sensitive to and indicative of potential boundary pixels and/or edge pixels in each 
analyzed source image. A wide variety of techniques usable in any known or later- 
developed edge-detection method are usable in various exemplary embodiments of the 
systems and methods according to this invention, as will be described further below. 

25 In various other embodiments, any known or later-developed edge detector, such as 
the Canny edge detector (the structure and operation of which is well-known to those 
of ordinary skill in the art) and the like, may be used in its entirety in step S600 to 
provide results usable in the systems and methods according to this invention. Next, 
in step S700, the results of the edge and/or boundary analysis technique are evaluated 

30 to determine a set of "edge" pixels in the composite image, as will be described further 
below. Operation then continues to step S800. 
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[0046] In step S800, each of the plurality of source images is characterized 
and/or analyzed using a technique that is sensitive to and indicative of potential 
surface pixels in each analyzed source image. In the broadest sense under various 
exemplary embodiments of the systems and methods according to this invention, 
5 surface pixels includes any pixels that are not determined to be edge and/or boundary 
pixels. A wide variety of techniques usable in any known or later-developed surface- 
focus detection method, region characterization method, and/or image segmentation 
method, are usable in various exemplary embodiments of the systems and methods 
according to this invention, as will be described further below. Next, in step S900, for 

1 0 pixels remaining undetermined in the composite image, that is, for those pixels of the 
composite image which have not been previously determined, the results of the 
surface pixel analysis technique are used to determine a set of "surface" pixels in the 
composite image. In various exemplary embodiments, region-filling or region- 
growing type operations are used to determine the composite image surface pixels 

15 according to the systems and methods of this invention, as described in detail further 
below. Operation then continues to step SI 000. 

[0047] In step S 1 000, any voids in the composite image representation 
resulting from the previous steps are filled. A variety of techniques are known for 
filling such voids in conventional images. Any known or later-developed void filling 

20 technique may be used in various exemplary embodiments of the systems and 

methods according to this invention. However, because there are a plurality of source 
images which may potentially be used as the basis for the pixels values to be used to 
fill the voids in the composite image, in various exemplary embodiments, the 
techniques used to fill the voids are supplemented with additional operations for 

25 selecting which source images will provide that basis, as will be described further 
below. Operation then continues with step SI 100. 

[0048] It should be appreciated that step S 1 000 is optional, and thus may be 
omitted, depending on the details of the edge and/or boundary and/or the surface 
analyses used in various embodiments of the systems and methods according to this 

30 invention. If the edge and/or boundary and/or the surface analyses inherently 

determine all pixels of the composite image, then inspecting the composite image 
representation for voids and filling the voids is not strictly necessary. In this case, step 
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SI 000 is omitted, and operation simply skips from step S900 to step SI 100. 
However, if the previous steps do not determine all pixels of the composite image, 
then it is preferred to fill the voids in the composite image representation using step 
SI 000 

5 [0049] In step S 1 1 00, the composite image representation is then stored to 

a memory device and/or output to a display device or recording device in any of a 
variety of suitable forms. In various exemplary embodiments, this storing and/or 
output step may include operations for transforming the composite image 
representation resulting from the previous steps into a subsequent electronic or 

1 0 physical representation of the composite image which is more convenient, desirable 
and/or useful for a particular purpose. For example, depending on the details of the 
previous steps, in various embodiments of the systems and methods according to this 
invention, separately stored records of the edge-pixels and the surface-pixels of the 
composite image are assembled or merged into a single stored representation of the 

15 composite image. 

[0050] As a further example, in various embodiments of the systems and 
methods according to this invention, the composite image representation may be in the 
form of a "map" of the spatial locations in the composite image, where the map 
contains function values or indexes, in contrast to actual pixel intensity values. 

20 Furthermore, the spatial resolution in all or part of the map may be at the spatial 

resolution of the pixels in the source images, or it may at a coarser spatial resolution in 
the form of "meta-pixels". Thus, in such embodiments, before the corresponding 
composite image can be meaningfully displayed as a conventional image, 
conventional gray-scale or color image data must be assembled by transforming such a 

25 "map" representation to the desired resolution and data format. In any case, in step 

SI 100, a desired representation of the composite image which is convenient, desirable 
and/or useful for a particular purpose is stored to a memory device and/or output to a 
display device or recording device. Operation then continues to step S 1200, where the 
operation ends. 

3 0 [0051] It should be appreciated that steps S 1 00 - S500 are just one 

exemplary embodiment of the method for providing a plurality of source images 
usable according to the systems and methods of this invention. Thus, it should be 
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appreciated that the systems and methods of this invention generally, and steps S600 - 
SI 200 in particular, are useable with any similar plurality of source images, regardless 
of the details of the underlying imaging system, the portion of the electromagnetic 
spectrum used to acquire the plurality of source images, or the particular operations 
5 used to acquire and/or prepare a similar plurality of source images. 

[0052] With regard to the terms "boundaries" and "edges" as used herein, 
the terms "boundaries" and "edges" are generally used interchangeably with respect to 
the scope and operations of the systems and methods of this invention. However, 
when the context dictates, the term "edge" may further imply the edge at a 

10 discontinuity between different surface planes on an object and/or the image of that 
object. Similarly, the term "boundary" may further imply the boundary at a 
discontinuity between two colors, or other relatively homogeneous surface properties, 
on a relatively planar surface of an object, and/or the image of that object. 
[0053] The terms "boundary pixels" and "edge pixels" are used 

1 5 interchangeably with respect to the scope and operations of the systems and methods 
of this invention. Both terms are intended to refer to pixels in the vicinity of local 
intensity changes in an image. These local intensity changes can be either grayscale 
intensity or a color intensity, for example. These local intensity changes are generally 
detectable by any conventional or enhanced, or any later-developed, edge-detection 

20 operation. Similarly, the terms "boundary properties" and "edge properties" are used 
interchangeably with respect to the scope and operations of the systems and methods 
of this invention. Both terms are intended to refer to any property, whether 
numerically characterizable or otherwise, that is usable, either alone or in 
combination, by any conventional or enhanced, or any later-developed, edge-detection 

25 operation. 

[0054] Some operations usable to characterize boundary properties and to 
detect edges include, but are not limited to, high-frequency spatial filtering, gradient 
analyses, directional gradient analyses, gradient derivative analyses, aspect ratio 
analysis, extent and/or connectedness analysis of candidate edges, and the like. 
30 Gradient analysis includes, but is not limited to, any one or more of gray-scale 
intensity, color intensity, texture measures, and the like. Aspect ratio analysis 
includes, but is not limited to, evaluating the length vs. width of candidate edges. 
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[0055] These and other operations which are sensitive to and indicative of 
boundary and/or edge properties are known to one skilled in the art of image 
processing, and are included in one or more of a wide variety of known edge-detection 
techniques. The chapter regarding edge-detection in the Machine Vision, by Ramesh 
5 Jain, et al., McGraw Hill, 1995 discusses related teachings and references. It should 
also be appreciated that nearly all of the operations discussed above which are 
sensitive to and indicative of boundary properties and usable to identify edges in an 
image are furthermore sensitive to and indicative of the degree to which an edge is 
focused in an image, since out-of- focus edges are blurred and therefore exhibit a 

10 different spatial distribution of intensities than when focused, and, when extremely 
out-of- focus, are not visible at all. 

[0056] It should be further appreciated that, in various exemplary 
embodiments, it is desirable to use an edge-detection operation which operates over 
all of the analyzed source images to determine the set of composite image edge pixels. 

15 This set of composite image edge pixels will include an edge pixel at any spatial 

location where analysis of the plurality of spatially congruent source images indicates 
that there is a valid edge pixel in at least one of the spatially congruent source images. 
Accordingly, it should be appreciated that, in various embodiments of the systems and 
methods according to this invention, the operations previously described 

20 corresponding to steps S600 and S700 combine to implement an edge-detection 
operation which operates over the plurality of source images to perform a type of 
analysis of the source images usable to determine a first set of pixels of the composite 
image corresponding to edges and/or boundaries in the composite image. 

[0057] Thus, the operations previously described as corresponding to steps 

25 S600 and S700 might alternatively be characterized as a single, more complex, multi- 
image composite edge-detection operation corresponding to a single step. 
Accordingly, in various embodiments, any known or later-developed multi-image 
composite edge detector, such as the multi-image fusion edge detector disclosed in the 
previously discussed '449 Patent, may be used to provide results usable in the systems 

30 and methods according to this invention. Various other exemplary operations related 
to determining edge pixels in the composite image are discussed in detail further 
below. 
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[0058] With regard to the term "surface pixels" as used herein, as 
previously mentioned, in the broadest sense according to this invention, "surface 
pixels" include any pixels that are not determined to be boundary pixels and/or edge- 
pixels. In addition, in many practical cases according to the systems and methods of 
5 this invention, "surface pixels" will be pixels that constitute an image of a relatively 
well-focused, homogenous, and relatively planar local region on an object. The term 
"surface property", as used herein, generally refers to any property, whether 
numerically-characterizable or otherwise, that is usable, either alone or in 
combination, by any known or later-developed region characterization operation or 

10 technique, such as, for example, surface and texture classification methods. 

[0059] Stated another way, surface properties are the properties associated 
with the statistical properties and spatial distribution of gray levels and/or color levels 
in a particular local neighborhood. Some operations usable to characterize surface- 
properties include but are not limited to, for example, variance analysis, average 

15 intensity, other statistical texture and/or contrast measures, fractal dimension, texture 
directionality measures, and the like. Variance analysis and average intensity can each 
be applied to, for example, gray level values or color level values. 

[0060] These and other operations which are sensitive to and indicative of 
surface properties are known to one skilled in the art of image processing, and are 

20 included in one or more of a wide variety of known surface-focus detection (auto- 
focus) methods, region/texture characterization methods, and/or image segmentation 
methods. The chapters regarding texture and regions in the Machine Vision, by 
Ramesh Jain, et al., McGraw Hill, 1995 discuss related teachings and references. It 
should be appreciated that nearly all of the operations discussed above which are 

25 sensitive to and indicative of surface properties for the purpose of classifying surface 
regions, and especially those useful for the purpose of classifying textures, are 
furthermore sensitive to and indicative of the degree to which a surface is focused in 
an image, since out-of-focus surface textures are blurred and therefore exhibit a 
different statistical and spatial distribution of intensities than when focused, and when 

30 extremely out-of-focus are show only a uniform intensity (i.e., perfectly smooth 
texture). 
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[0061] With regard to the determination of surface pixels in the composite 
image, it should be appreciated that, in various exemplary embodiments, it is desirable 
to use a surface-pixel determining operation which operates over all of the analyzed 
source images to determine a set of composite surface pixels including any spatial 
5 location which is not otherwise determined to be the location of a boundary pixel 
and/or an edge pixel. Accordingly, it should be appreciated that, in various 
embodiments of the systems and methods according to this invention, the operations 
previously described as corresponding to steps S800 and S900 combine to implement 
a surface-detection operation which operates over the plurality of source images to 

10 perform a type of analysis of the source images usable to determine a set of pixels of 
the composite image corresponding to surface regions in the composite image. Thus, 
the operations previously described as corresponding to steps S800 and S900 might 
alternatively be characterized as a single, more complex, multi-image composite 
surface-detection operation corresponding to a single step. Various exemplary 

1 5 operations related to determining surface pixels in the composite image are discussed 
in detail further below. 

[0062] Fig. 4 is a flowchart outlining in greater detail one exemplary 
embodiment of analyzing the source images using a technique sensitive to and 
indicative of potential boundary pixels and/or edge pixels of step S600 of Fig. 3, 

20 according to this invention. Beginning in step S600, operation continues to step S610, 
where the source images to be analyzed are determined or defined manually, in a 
semi-automated fashion, or fully automatically. For example, in the general case, one 
or more sets of source images may have been previously acquired and stored. In such 
a case, step S600 may start after a long interruption of operations. Thus, it is 

25 necessary to select and recall a particular set of source images. In another case, a set 
of source images may include images which include no focused features in the desired 
composite image region. It may be desirable to eliminate such source images from 
analysis, to reduce processing time. 

[0063] Next, in step S620, one of the source images determined in step 

30 S61 0 which has not been previously selected for analysis is selected as the current 

source image to be analyzed. Then, in step S630, the data is accessed for the current 
source image to be analyzed. More particularly, the data is accessed for the spatial 
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locations within the current source image which correspond to the spatial locations to 
be included in the desired composite image. Operation then continues to step S640. 

[0064] In step S640, the accessed source image data is analyzed to provide 
edge-focus indicator results and to provide a local result at a desired spatial resolution. 
5 In various exemplary embodiments, in step S640, the accessed source image data is 
analyzed using a mathematical or algorithmic technique which is sensitive to and 
indicative of local intensity changes in the image. These local intensity change can be 
either changes in the grayscale intensity values or changes in the color intensity 
values, for example. 

1 0 [0065] In various exemplary embodiments, the data for the current source 

image is first normalized with respect to white level. Then, two gradient operators 
which are the derivatives of a 2-dimensional Gaussian in the directions of the rows 
and columns of the image data, respectively, are defined. In one exemplary 
embodiment the 2-dimensional Gaussian is set at a size of 1 1 by 1 1 pixels , and an 

1 5 effective width of 9 pixels. These gradient operators are systematically applied at 
each possible pixel location throughout the current source image, according to well- 
known convolution methods. The magnitude of the sum of the squares of the 
respective gradient operator results is taken as the local gradient result, for each pixel 
throughout the current source image. In this case, the local gradient results are the 

20 edge-focus indicator results. Such techniques are known to one skilled in the art of 
image processing. Machine Vision, by Ramesh Jain, et al., McGraw Hill, 1995 
discusses related teachings. 

[0066] Then, in step S650, the edge-focus indicator results are combined 
into meta-pixels. Next, in step S660, an edge-focus indicator result representative of 

25 each meta-pixel is determined. The edge-focus indicated result can be determined, for 
example, by averaging all of the edge-focus indicator results encompassed by the 
meta-pixel. Typically, if meta-pixels are used with respect to one source image, then 
meta-pixels should be used with respect to each source image. Operation then 
continues to step S670. 

30 [0067] It should be appreciated that steps S650 and S660 are optional, 

depending, for example, on the magnification and resolution of the source images 
relative to the features on the imaged object, and/or whether it is desirable to 
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minimize the composite image processing time. If the composite image processing 
time is not important in a particular application, then operation simply skips from step 
S640 directly to step S670. 

[0068] In step S670, the determined edge-focus indicator results are 
5 recorded or stored for subsequent use. It should be appreciated that, regardless of the 
spatial resolution, that is, whether at a pixel a pixel resolution or at a meta-pixel 
resolution, the edge-focus indicator results for the current source image encompass, 
and are generally spatially congruent with, the spatial locations included in the desired 
composite image. Then, in step S680, a determination is made whether all the 

1 0 determined source images have been analyzed. If there are more source images to be 
analyzed then operation returns to step S620. Otherwise, operations corresponding to 
step S600 of Fig. 3 are complete. In this case, operation continues to step S690, 
where the operation continues to step S700. 

[0069] Fig. 5 is a flowchart outlining in greater detail one exemplary 

1 5 embodiment of determining or selecting a set of edge pixels of step 700 of Fig. 3 

according to this invention. As indicated above, in various exemplary embodiments, 
the determination or selection of the edge pixels is based on the results from an 
analysis technique which is sensitive to and indicative of potential boundary pixels 
and/or edge pixels. Beginning in step S700, operation continues to step S705, where 

20 the source image analysis results to be analyzed are determined. Then, in step S710, 
the spatial locations to be analyzed in the determined source image analysis results are 
identified. Operation then continues to step S715. 

[0070] Depending on the particular structure of the composite image 
processing according to this invention, it may be necessary to establish or reestablish 

25 which source image analysis results are to be further analyzed. These source image 
analysis results to be further analyzed may be determined or defined manually, in a 
semi-automated fashion, or fully automatically. In various exemplary embodiments, 
the source image analysis results determined or defined in a previous step, such as 
step S600 of Fig. 3, represent the source image analysis results to be further analyzed. 

30 [0071] Similarly, depending on the particular structure of the composite 

image processing according to this invention, it may be necessary to establish or 
reestablish which spatial locations in the source image analysis results are to be 
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further analyzed. In general, these spatial locations correspond to the spatial locations 
included in the desired composite image. These may be determined or defined 
manually, in a semi-automated fashion, or fully automatically. In various exemplary 
embodiments, the source image analysis results determined or defined in a previous 
5 step, such as step S600 of Fig. 3, represent the spatial locations to be further analyzed. 

[00721 In step S7 1 5, one of the spatial locations identified in step S7 1 0 
which has not been previously selected for analysis according to this step S715 is 
selected as the current spatial location to be analyzed. Next, in step S720, the source 
image analysis results data corresponding to each source image identified in step 

10 S705, are accessed for the current spatial location to be analyzed. Stated another way, 
all of the spatially congruent source image analysis results are accessed for all source 
images, for a current spatial location. 

[0073] Then, in step S725, the accessed data is analyzed to determine or 
select the best source image analysis result, to maximize the composite image focus 

15 results. For example, if the source image analysis results are numerical edge-focus 
indicator results, a most appropriate one of the plurality of numerical edge-focus 
indicator results is selected. In various exemplary embodiments, the most appropriate 
numerical edge-focus indicator result is the maximum result. Operation then 
continues to step S730. 

20 [0074] In step S730, a determination is made whether the selected source 

image analysis result is indicative of a valid edge. For example, if the source image 
analysis result is a numerical edge-focus indicator result, this result is compared to a 
numerical threshold value, where only values exceeding the threshold value are taken 
to be indicative of a valid and adequately-focused edge for purposes of constructing 

25 the composite image. If the source image analysis result is not indicative of a valid 
edge, then operation returns to step S715. Otherwise, operation continues to step 
S735. 

[0075] In an alternative embodiment of the operations described relative to 
steps S725 and S730, processing time is minimized by selecting the first processed 
30 source image analysis result which is indicative of a valid and adequately-focused 
edge for purposes of constructing the composite image. For example, if the source 
image analysis results are the previously-discussed numerical edge-focus indicator 
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results, the plurality of numerical edge-focus indicator results is systematically 
compared to a numerical threshold value indicative of a valid and adequately-focused 
edge. Then, the first result to exceed the threshold is selected as a source image 
analysis result for constructing the composite image. In this case, following the 
5 selection of a result exceeding the threshold, operation continues to step S735. 
Otherwise, if no source image analysis result is indicative of a valid edge, then 
operation returns to step S715. 

[0076] In step S735, the determined source image analysis result indicative 
of a valid edge, and/or an indication of the corresponding source image, such as an 

1 0 index number, is recorded or stored relative to the current spatial location. It should 
be appreciated that it is convenient to refer to the data elements and their respective 
associations recorded in step S735 as elements of a "composite edge map". However, 
it should be understood that whenever the term "map" is used herein, the term is being 
used only as a convenient reference for the underlying data elements and their 

1 5 respective associations, regardless of whether these recorded data elements are ever 
actually structured into a map in any conventional sense. It should be further 
understood that the scope of the systems and methods of this invention is in no way 
restricted to any particular representation of the basic underlying data and data 
associations disclosed herein. 

20 [0077] Then, in step S740, a determination is made whether all of the 

previously determined spatial locations to be analyzed have been analyzed. If there 
are more spatial locations to be analyzed, then operation returns to step S715. 
Otherwise, operation continues to step S745. 

[0078] In step S745, a determination is made whether to modify or refine 

25 the data in the composite edge map. The inventor has determined that with or without 
modifying the composite edge map, the systems and methods of this invention 
usefully suppress artifacts in a wide variety of composite images. However, as 
discussed in detail further below, the composite edge map can also be refined by 
various operations to further suppress artifacts in the resulting composite image. 

30 Thus, both alternatives are within the scope of the systems and methods of this 

invention. The decision to modify or refine the composite edge map data may depend, 
for example, on the magnification and/or resolution of the source image relative to the 
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features on the imaged object. These factors influence the level of detail visible in the 
source images, as well as the size of artifacts that may be present in out-of-focus 
image portions. 

[0079] This decision may also or instead depend on, for example, whether 
5 the composite image processing time should be minimized. Depending on the 
particular structure of the composite image processing system and/or method 
according to this invention, this decision may be made manually, in a semi-automated 
fashion, or fully automatically. If the composite image processing time is not 
important in a particular application, then, as a default condition it may be decided to 

10 modify or refine the composite edge map. Alternatively, if an operator has previously 
observed similar composite images of similar objects, as may typically happen in a 
repetitive manufacturing or inspection environment, the operator may make the 
decision at least in part based on the operator's previous observations. If it is decided 
not to modify or refine the composite edge map, then operation jumps to step S770. 

1 5 Otherwise, if the composite edge map is to be modified or refined, operation 
continues to step S750. 

[0080] In step S750 the composite edge map is refined. That is, the data 
associated with spatial locations corresponding to the composite image, and more 
particularly, the data associated with spatial locations in and around the spatial 

20 locations corresponding to valid edge results in the previously determined composite 
edge map, is modified through one or more operations to further suppress artifacts in 
the resulting composite image and/or to otherwise enhance the resulting composite 
image. A variety of well-known techniques used in known edge-detection methods, 
as well as any appropriate later-developed techniques, are usable according to various 

25 embodiments of the systems and methods according to this invention. For example, 
techniques based second-derivative operators may be used to "thin" edges in the 
composite edge map, thus removing some "blur" and other artifacts adjacent to edges. 
Additionally or alternatively, weak edges and fragments which are likely artifacts may 
be detected and eliminated by these methods. Such techniques are known to one 

30 skilled in the art of image processing. Machine Vision, by Jain, et al., discusses 
related teachings. After the data in the composite edge map is refined, operation 
continues to step S770. 
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[0081] In step S770, depending on the particular structure of the composite 
image processing system and/or method according to this invention, the data in the 
composite edge map, whether refined or not, is assembled, and then stored and/or 
output, as a representation usable to determine the edge pixels in the composite image. 
5 It should be appreciated that, regardless of the spatial resolution, that is, whether at 
pixel or meta-pixel resolution, this representation of the edge pixels in the composite 
image encompasses, and is generally spatially congruent with, the spatial locations 
included in the desired composite image. As shown in Fig. 5, operations 
corresponding to step S700 of Fig. 3 are complete at this point. In this case, operation 

10 continues to step S775, where operation continues to step S800. 

[0082] Fig. 6 is a flowchart outlining in greater detail one exemplary 
embodiment of a set of operations usable for refining the composite edge map of step 
S750 of Fig. 5 according to this invention. Beginning in step S750, operation 
continues to step S751, where , depending on the particular structure of the composite 

1 5 image processing system and/or method according to this invention, it may be 

necessary to access or re-access the data previously referred to as the composite edge 
map, so that it may be modified and/or refined in subsequent steps. Operation then 
continues with step S752. 

[0083] In step S752, the data referred to as the composite edge map is 

20 overruled and/or refined for spatial locations where "weak" edge-indicating data 

occurs within a prescribed distance of "strong" edge-indicating data. A strong edge, 
such as, for example, an edge which leads to a large numerical value in a gradient 
analysis, which is partially out of focus in a first one of the source images, may still be 
analyzed as an edge, although it is better-focused and more realistically imaged in a 

25 different one of the source images. When the strong edge is out of focus, the strong 
edge may be detected in a displaced or broaden location. This tends to produce 
artifacts if the strong edge is admitted to the composite image in an out-of-focus state. 
In step S752, at least some such artifacts not otherwise eliminated by the systems and 
methods of this invention are suppressed or eliminated. 

30 [0084] In an exemplary embodiment, the edge-indicating data is in the form 

of the previously discussed gradient results combined into 5x5 meta-pixels. Each 
spatial location in the data referred to as the composite edge map is systematically 
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analyzed to remove the edge-indicating data or to set it to zero at any spatial location 
where the original value of the edge-indicating data at that spatial location is less than 
the average of its 4-connected spatial neighbors. 

[0085] Then, in step S753, the particular source images and/or focal planes 
5 represented in the current composite edge map are determined. In a first exemplary 
embodiment, each unique source image index represented in the data of the current 
composite edge map is recorded for governing subsequent iterative operations. In a 
second exemplary embodiment, the indexes of the entire plurality of source images 
are used for governing subsequent iterative operations. Next, in step S754, one of the 

1 0 source images determined in step S753 which has not been previously selected for 

analysis is selected as the current source image to be analyzed. In various exemplary 
embodiments, the current source image is identified by its corresponding source image 
index. Operation then continues to step S755. 

[0086] In step S755, the data of the current composite edge map which 

1 5 corresponds to the current source image index is accessed or identified for evaluation. 
That is, all spatial locations in the current composite edge map which correspond to 
the current underlying source image are to be operated upon in the present iteration. 

[0087] Next, in step S756, all spatial locations in the current composite 
edge map which correspond to the current underlying source image are analyzed to 

20 eliminate edge-pixels which are not part of an extended "line" or a spot. This further 
qualifies and cleans up the edges included in the data corresponding to the current 
composite edge map. In one exemplary embodiment, the analysis is done with 
reference to the previously described combined 5x5 meta-pixels, that is, with 
reference to "edge meta-pixels". Each spatial location corresponding to an edge meta- 

25 pixel is systematically analyzed. Each 5x5 edge meta-pixel is required to be part of a 
local group of four or more edge meta-pixels, where each meta-pixel of the local 
group must touch at least one other 4-connected neighbor in the local group. For 
spatial locations where isolated edge meta-pixels fail this criterion, the corresponding 
edge-indicating data and/or source image index is removed or set to zero. Operation 

30 then continues with step S757. 

[0088] In an alternative exemplary embodiment, the previous embodiment 
is used except that each meta-pixel of the local group must touch at least one other 8- 



May25,2001 



MEIRef.: 2000-04 
O&BRef.: 106538 



28 

connected neighbor in the local group. In yet another alternative exemplary 
embodiment, each spatial location corresponding to an edge meta-pixel is 
systematically eroded with four structuring elements. Each structuring element is one 
meta-pixel wide and is four meta-pixels long in one of the horizontal, vertical and 
5 each diagonal direction, respectively. Following the erosion operation, each spatial 
location corresponding to an edge meta-pixel is systematically dilated with the same 
four structuring elements. This combination of operations is sometimes known as 
"opening" in the field of image processing. Such techniques are known to one skilled 
in the art of image processing. Machine Vision, by Ramesh Jain, et al., McGraw Hill, 

10 1995 discusses related teachings. For the eliminated meta-pixels, the corresponding 
edge-indicating data and/or source image index is removed or set to zero in the data 
corresponding to the composite edge map. 

[0089] The previous operations of steps S75 1 -S756 have been directed 
toward further refining the data corresponding the to the composite edge map, such 

15 that edge artifacts and noise corresponding to the current underlying source image are 
further suppressed. The composite edge map data remaining after step S756 of the 
current iteration is likely to corresponds to "true" edges that are reasonably well- 
focused in the current underlying source image. 

[0090] In step S757, the set of spatial locations corresponding to the current 

20 underlying source image in the current composite edge map as modified in step S756, 
is spatially "expanded" to add adjacent edge pixels to the current composite edge map. 
Such added adjacent pixels will broaden the edge images a bit, and are selected in step 
S757 from the same source image as the adjacent well-focused edge. Thus, a good 
composite image should result. In one exemplary embodiment, the set of previously 

25 described binned 5x5 edge meta-pixels corresponding to the current underlying source 
image are systematically operated on with a 3x3 structuring element to perform a 
dilation. 

[0091] Next, in step S758, for each spatial location where an added edge 
meta-pixel is obtained in step S757, the corresponding edge-indicating data and/or 
30 source image index is recorded in the data corresponding to the composite edge map. 

[0092] Then, in step S759, a determination is made whether all of the data 
corresponding to all of the previously determined source images and/or source image 
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indexes have been analyzed. If there is data remaining to be analyzed then operation 
returns to step S754, otherwise operation continues at step S760. 

[0093] As a result of the previous operations of steps S75 1-S759, the data 
corresponding to all source images underlying the current composite edge map have 
5 been analyzed, refined and the refined results recorded. It should be appreciated that, 
as a result of these previous operations, it may occur that, for some spatial locations to 
be included in the composite image, more that one underlying source image is 
indicated to correspond to a valid well-focused edge. Thus, in step S760, for each 
spatial location having any corresponding edge-indicating data and/or source image 

10 index recorded in the data corresponding to the composite edge map, a determination 
is made regarding which underlying source image corresponds to the best edge image. 
For example, in various exemplary embodiments of step S760, this determination is 
made by identifying the maximum- valued edge-indicating data. Furthermore, in step 
S760, the corresponding best edge-indicating data and/or corresponding source image 

15 index is recorded in the data corresponding to the refined composite edge map as the 
basis for determining the composite image. Operation then continues to step S761, 
where the operation continues to step S770 of Fig. 5. 

[0094] Fig. 7 is a flowchart outlining in greater detail one exemplary 
embodiment of analyzing the source images using a technique sensitive to and 

20 indicative of potential surface pixels of step S800 of Fig. 3 according to this invention. 
Beginning in step S800, operation continues to step S810, where the source images to 
be analyzed are determined or defined in a manual, semi-automated, or automated 
fashion. For example, in the general case, one or more sets of source images have 
been previously acquired and stored. In such a case, step S800 may start after a long 

25 interruption of operations. Thus, it may be necessary to select and recall a particular 
set of source images. In another case, a set of source images may include images 
which include no focused features in the desired composite image region. Thus, in 
this case, it may be desirable to eliminate such source images from analysis to reduce 
processing time. 

30 [0095] Next, in step S820, one of the source images determined in step 

S810 which has not been previously selected for analysis is selected as the current 
source image to be analyzed. Then, in step S830, the data is accessed for the current 
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source image to be analyzed. More particularly, the data is accessed for the spatial 
locations within the current source image which correspond to the spatial locations 
included in the desired composite image. Operation then continues to step S840. 

[0096] In step S840, the accessed source image data is analyzed to provide 
5 surface-focus indicator results and to provide a local result at a desired spatial 

resolution. In various exemplary embodiments, in step S840, the accessed source 
image data is analyzed using a mathematical or algorithmic technique which is 
sensitive to and indicative of local surface focus in the image. The local surface focus 
can be indicated, for example, by the spatial distribution of either grayscale intensities 

10 or color intensities, in a local neighborhood. Operation then continues to step S850. 

[0097] In various exemplary embodiments of step S840, the data for the 
current source image is systematically operated upon such that, at each possible spatial 
location, an average image value, for example a gray level, is determined from the all 
the image values included in a 5x5 square of spatial locations centered on that spatial 

1 5 location. The resulting average local image value is subtracted from the image data 
value at that spatial location to create a residual value for each corresponding spatial 
location. For convenience, the resulting set of residual values and corresponding 
spatial locations is herein called a "residual image". 

[0098] Next, in various exemplary embodiments, for each spatial location, 

20 the standard deviation of the residual values in a 5x5 region centered on that spatial 
location is determined. The standard deviation is divided by the mean of the residual 
values in that region to determine a surface-focus indicator result for that spatial 
location. For the surface-focus indicator results of the foregoing exemplary 
embodiments, a low value for the indicator corresponds to a smooth or unfocused 

25 surface region. In contrast, a high value for the indicator corresponds to an in-focus 

textured region. Many other techniques and metrics indicative of well-focused surface 
regions are known to one skilled in the art of image processing. Such techniques may 
be used as alternatives to the foregoing exemplary embodiments, to provide surface 
focus indicators usable according to the systems and methods of this invention. 

30 Machine Vision, by Ramesh Jain, et al., McGraw Hill, 1995 discusses related 
teachings. 
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[0099] In step S850, the surface-focus indicator results are combined into 
meta-pixels. Next, in step S860, a surface-focus indicator result representative of 
each meta-pixel is determined. The surface-focus indicator result can be determined, 
for example, by averaging all of the surface-focus indicator results encompassed by 
5 the meta-pixel. Typically, if meta-pixels are used with respect to one source image, 
then meta-pixels should be used with respect to each source image. It should be 
appreciated that, by matching the combined surface meta-pixel size to a previously 
used or prescribed edge meta-pixel size, and by further insuring that the combined 
surface meta-pixels and the combined edge meta-pixels spatially align in a one-to-one 

1 0 correspondence, computations and record keeping according to the systems and 

methods of this invention can be simplified. Operation then continues to step S870. 

[0100] It should be appreciated that steps S850 and S860 are optional, 
depending on, for example, the magnification and resolution of the source images 
relative to the features on the imaged object, and/or whether it is desirable to 

1 5 minimize the composite image processing time. If the composite image processing 
time is not important in a particular application, then operation simply skips steps 
S850 and S860 and continues directly from step S840 to step S870. 

[0101] It should be further appreciated that, in various exemplary 
embodiments, steps S840-S860 may be effectively combined and/or may be 

20 indistinguishable. For example, in an alternative exemplary embodiment, the residual 
image is determined as described above with regard to step S830. The residual image 
is then divided into a regular array of 5x5 regions. For each 5x5 region, the standard 
deviation of the residual values in that region is determined The determined standard 
deviation is divided by the mean of the residual values in that region to determine a 

25 surface-focus indicator result representative of that entire region and/or spatial 

location. In this alternative exemplary embodiment, it should be noted that the 5x5 
region size is chosen to correspond to the 5x5 edge meta-pixel size discussed in 
association with the exemplary edge-pixel operations described above. In this 
alternative exemplary embodiment, steps S840 and S850 of determining a surface- 

30 focus indicator result at one spatial resolution and then combining the result at another 
spatial resolution, are effectively merged. 
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[0102] In step S870, the determined surface-focus indicator results are 
recorded or stored for subsequent use. It should be appreciated that, regardless of the 
spatial resolution, that is, whether at a pixel resolution or at a meta-pixel resolution, 
the surface-focus indicator results for the current source image encompass and are 
5 generally spatially congruent with, the spatial locations included in the desired 
composite image. Then, in step S880, a determination is made whether all the 
determined source images have been analyzed. If there are more source images to be 
analyzed, then operation returns to step S820. Otherwise, operation continues to step 
S890, where the operation continues to step S900. 

10 [0103] Fig. 8 is a flowchart outlining in greater detail one exemplary 

embodiment of operations for determining or selecting a set of surface pixels in step 
S900 of Fig. 3 according to this invention As indicated above, in various exemplary 
embodiments, the determination or selection of the surface pixels is based on the 
results from an analysis technique which is sensitive to and indicative of potential 

15 surface pixels. Beginning in the step S900, operation continues to step S905, where 
the source image analysis results to be analyzed are determined. Depending on the 
particular structure of the composite image processing according to this invention, it 
may be necessary to establish or reestablish which source image analysis results are to 
be further analyzed. These source image analysis results to be further analyzed may 

20 be determined or defined manually, in a semi-automated fashion, or fully 

automatically. In various exemplary embodiments, the source image analysis results 
determined or defined in a previous step, such as step S800 of Fig. 3, represent the 
source image analysis results to be further analyzed. Operation then continues to step 
S910. 

25 [0104] In step S91 0, a determination is made whether to perform additional 

operations to further suppress potential edge-artifacts in the composite image. The 
inventor has determined that, with or without such additional operations, the systems 
and methods of this invention usefully suppress artifacts in a wide variety of 
composite images. However, the operations discussed below with regard to step S915 

30 can further suppress image artifacts which might otherwise appear in the composite 
image and which correspond to out-of-focus edges or boundaries included in the 
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plurality of source images. Thus, both alternatives are within the scope of the systems 
and methods of this invention. 

[0105] The decision to further suppress potential edge-artifacts in the 
composite image may depend on, for example, the magnification and/or resolution of 
5 the source image relative to the features on the imaged object. These factors influence 
the level of detail visible in the source images, as well as the size of artifacts that may 
be present in out-of-focus image portions. This decision may also or instead depend 
on, for example, whether the composite image processing time should be minimized. 
Depending on the particular structure of the composite image processing system 

10 and/or method according to this invention, the decision may be made manually, in a 
semi-automated fashion, or fully automatically. 

[0106] If the composite image processing time is not important in a 
particular application, then, as a default condition, it may be decided to further 
suppress potential edge-artifacts in the composite image. Alternatively, if an operator 

15 has previously observed similar composite images of similar objects, as may typically 
happen in a repetitive manufacturing or inspection environment, the operator may 
make the decision at least in part based on the operator's previous observations. If it is 
decided not to further suppress potential edge-artifacts in the composite image, then 
operation jumps to step S920. If it is decided to further suppress potential edge- 

20 artifacts in the composite image, then operation continues with step S915. 

[0107] In step S915, operations are performed to facilitate further 
suppressing potential edge-artifacts during subsequent surface-pixel determination 
operations. Then, in step S920, the spatial locations to be analyzed in the determined 
source image analysis results are determined. 

25 [0108] In various exemplary embodiments of step S91 5, since strong edges 

can create artifacts in immediately adjacent regions of an image when such strong 
edges are out of focus, the data corresponding to the current composite edge map is 
analyzed to determine a set of spatial locations adjacent to the edges included in the 
current composite edge map. These spatial locations are treated as edge-artifact buffer 

30 zones during subsequent surface-pixel determination operations. In one exemplary 
embodiment, the data corresponding to the current composite edge map corresponds 
to the previously described 5x5 combined edge meta-pixels. Operations are 
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performed which are essentially equivalent to dilating the spatial locations 
corresponding to the current composite edge map using a 5x5 structuring element, that 
is, 5x5 in terms of meta-pixels. Methods of dilating a set of spatial locations are 
known to one skilled in the art of image processing. Machine Vision, by Jain, et al., 
5 discusses related teachings. 

[0109] For the added spatial locations corresponding to the results of the 
dilation operation, that is, excluding the previous elements already included in the 
composite edge map, special data and/or values are assigned and/or recorded. The 
special data and/or values identify these spatial locations as comprising edge-artifact 

10 buffer zones during further surface-pixel determining operations. Special rules are 

then applied to determine surface pixels in the edge-artifact buffer zone. For example, 
in an exemplary embodiment described in greater detail below, surface-pixel regions 
may "grow" into the edge-artifact buffer zone. However, a surface-pixel region is not 
allowed to start from a seed pixel in the edge-artifact buffer zone. 

1 5 [0110] It should be appreciated that the previously described edge-artifact 

buffer zones may be alternatively described and determined as zones generally 
surrounding each edge included in the current composite edge map. In various 
exemplary embodiments, each zone has an outer zone-boundary, which is generally 
located at a prescribed setback distance from the edge pixels that the zone surrounds. 

20 The prescribed setback distance from the edge is generally prescribed in units of 

pixels, meta-pixels, or any set of units usable according to the systems and methods of 
this invention. 

[0111] Depending on the particular structure of the composite image 
processing system and/or method according to this invention, it may be necessary to 

25 establish or reestablish in step S920 which spatial locations in the source image 
analysis results are to be further analyzed. In general, these spatial locations 
correspond to the spatial locations included in the desired composite image. These 
may be determined or defined manually, in a semi-automated fashion, or fully 
automatically. In general, these spatial locations correspond to the spatial locations 

30 included in the desired composite image. 

[0112] In various exemplary embodiments, the source image analysis 
results determined or defined in a previous step, such as in step S800 of Fig. 3, 
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represent the spatial locations to be further analyzed. If edge-artifact buffer zones 
have been determined as described below, then the spatial locations corresponding to 
those edge-artifact buffer zones are also included to be further analyzed. Furthermore, 
optionally, the spatial locations already determined corresponding to the composite 
edge map may be excluded from the spatial locations to be further analyzed. 

[0113] In step S925, one of the spatial locations determined in step S920 
which has not been previously selected for analysis is selected as the current spatial 
location to be analyzed. Next, in step S930, the source image analysis results data 
corresponding to each source image determined in step S905 are accessed for the 
current spatial location to be analyzed. Stated another way, all of the spatially 
congruent source image analysis results are accessed for all source images, for the 
current spatial location. Then, in step S935, the accessed data is analyzed to 
determine or select the best source image analysis result, to maximize the composite 
image focus results. For example, if the source image analysis results are numerical 
surface-focus indicator results, a most appropriate one of the plurality of numerical 
surface-focus indicator results is selected. In various exemplary embodiments, the 
most appropriate numerical surface-focus indicator result is the maximum result. 
Operation then continues to step S940. 

[0114] In step S940, a determination is made whether the selected source 
image analysis result is indicative of a valid surface. For example, if the source image 
analysis result is a numerical surface-focus indicator result, this result is compared to 
a numerical threshold value, where only values exceeding the threshold value are 
taken to be indicative of a valid and adequately-focused surface for purposes of 
constructing the composite image. If the source image analysis result is not indicative 
of a valid surface, then operation returns to step S925. Otherwise, operation continues 
to step S945. 

[0115] In step S945, the determined source image analysis result indicative 
of a valid surface, and/or an indication of the corresponding source image, such as an 
index number, is recorded or stored relative to the current spatial location. Then, in 
step S950, a determination is made whether all of the previously determined spatial 
locations to be analyzed have been analyzed. If there are more spatial locations to be 
analyzed, then operation returns to step S925. Otherwise, operation continues to step 
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S955. 

[0116] It should be appreciated that it is convenient to refer to the data 
elements and their respective associations recorded in step S945 as elements of a 
"composite surface map". However, it should be understood that whenever the term 
5 "map" is used herein, the term is being used only as a convenient reference for the 

underlying data elements and their respective associations, regardless of whether these 
recorded data elements are ever actually structured into a map in any conventional 
sense. It should be further understood that the scope of the systems and methods of 
this invention is in no way restricted to any particular representation of the basic 

1 0 underlying data and data associations disclosed herein. 

[0117] In step S955, a determination is made whether to modify or refine 
the data in the composite surface map. The inventor has determined that, with or 
without modifying the composite surface map, the systems and methods of this 
invention usefully suppress artifacts in a wide variety of composite images. However, 

15 as discussed in detail further below, the composite surface map can also be refined by 
various operations to further suppress artifacts in the resulting composite image. 
Thus, both alternatives are within the scope of the systems and methods of this 
invention. The decision to modify or refine the composite surface map data may 
depend on, for example, the magnification and/or resolution of the source image 

20 relative to the features on the imaged object. These factors influence the level of 

detail visible in the source images, as well as the size of artifacts that may be present 
in out-of-focus image portions. 

[0118] This decision may also or instead depend on, for example, whether 
the composite image processing time should be minimized. Depending on the 

25 particular structure of the composite image processing system and/or method 

according to this invention, this decision may be made manually, in a semi-automated 
fashion, or fully automatically. If the composite image processing time is not 
important in a particular application, then, as a default condition it may be decided to 
modify or refine the composite surface map. Alternatively, if an operator has 

30 previously observed similar composite images of similar objects, as may typically 
happen in a repetitive manufacturing or inspection environment, the operator may 
make the decision at least in part based on the operator's previous observations. If it is 
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decided not to modify or refine the composite surface map, then operation jumps 
directly to step S965. Otherwise, if the composite surface map is to be modified or 
refined, operation continues to step S960. 

[0119] In step S960, the composite surface map is refined. That is, the data 
5 associated with spatial locations corresponding to the composite image, and more 
particularly, the data associated with spatial locations in and around the spatial 
locations corresponding to valid surface results in the previously determined 
composite surface map, is modified through one or more operations to further 
suppress artifacts in the resulting composite image and/or to otherwise enhance the 

10 resulting composite image. Operation continues with step S965. 

[0120] A variety of known image-enhancement techniques, as well as any 
appropriate later-developed techniques, are usable in step S960 according to various 
embodiments of the systems and methods according to this invention. For example, 
in various previously described exemplary embodiments, where a numerical surface- 

1 5 focus indicator result is compared to a numerical threshold value in step S940 to 

determine valid surface pixels, the data corresponding to the composite surface map 
may, in some cases, include voids surrounded by valid surface pixels. In this case, 
known void-filling techniques may be used to fill such voids. Such techniques are 
known to one skilled in the art of image processing. Machine Vision, by Jain, et al, 

20 discusses related teachings. However, because there are a plurality of source images 
which may potentially be used as the basis for the pixels values to be used to fill a 
void in the composite image, known methods for void filling must be supplemented 
with an additional method for selecting which source image, or source images, will 
provide the basis for filling a void. 

25 [0121] In one exemplary embodiment of an additional method for selecting 

which source image, or source images, will provide the basis for filling a void, an 
erosion operation is followed by a dilation operation according to known techniques. 
Then, for each void which is filled according to these operations, the void is filled 
based on the source image index which is closest to the average of the source image 

30 indexes of the spatial locations which surround the void. In other exemplary 

embodiments, region-growing methods are used. Known region growing methods 
may be used, and such techniques are known to one skilled in the art of image 
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processing. Machine Vision, by Jain, et al., discusses related teachings. However, 
similarly to known void-filling methods, known region growing methods must be also 
be supplemented with an additional method for selecting which source image, or 
source images, will provide the basis for the pixels which will be added to the 
growing region. An exemplary embodiment of region-growing operations usable for 
refinement of the composite surface map is described with reference to Fig. 9, further 
below. 

[0122] In step S965, depending on the particular structure of the composite 
image processing system and/or method according to this invention, the data in the 
composite surface map, whether refined or not, is stored and/or output, as a 
representation usable to determine the surface pixels in the composite image. 
Operation continues to step S970, where operation continues to step SI 000. 

[0123] In various exemplary embodiments, the data corresponding to the 
composite surface map is combined with the data corresponding to the previously- 
determined composite edge map and/or composite image The combined surface and 
edge-pixel data is stored and/or output as a representation usable to determine the 
composite image. It should be appreciated that, regardless of the spatial resolution, 
that is, whether at pixel or meta-pixel resolution, this representation of the surface 
pixels and/or edge pixels in the composite image encompasses, and is generally 
spatially congruent with, the spatial locations included in the desired composite image 
each spatially congruent surface and/or edge pixel of the composite image may thus be 
determined at a pixel resolution from this representation. 

[0124] For example, in various exemplary embodiments, surface-pixel 
and/or edge-pixel data is represented in the form of surface- and/or edge-meta-pixels, 
respectively, in the composite image representation. Each surface and/or edge pixel 
that is spatially congruent with a particular surface and/or edge meta-pixel, 
respectively, is given an image value based on the corresponding spatially congruent 
pixel in the source image corresponding to the source image index of that surface- 
meta-pixel. 

[0125] Fig. 9 is a flowchart outlining in greater detail one exemplary 
embodiment of operations for performing the step of refining the composite surface 
map of step S960 of Fig. 8 according to this invention. Beginning in step S960, 
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operation continues to step S961, where, depending on the particular structure of the 
composite image processing system and/or method according to this invention, it may 
be necessary to access/re-access the current data representation of the composite 
surface map, so that the current data representation of the composite surface map may 
5 be modified and/or refined in subsequent steps. Operation then continues with step 
S962. 

[0126] In step S962, a determination is made whether operations have 
previously been performed to identify spatial locations comprising edge-artifact buffer 
zones, for example, as described above with respect to steps S910-S915 ofFig. 8. If it 

10 is determined that operations have not previously been performed to identify spatial 
locations comprising edge-artifact buffer zones, or if, in various exemplary 
embodiments, such previous operations are ignored, then operation jumps to step 
S964. Otherwise, if it is determined to use previously determined edge-artifact buffer 
zones in subsequent operations to further suppress potential edge-artifacts in the 

1 5 composite image, then operation continues with step S963 . 

[0127] In step S963, the current data representation of the edge-artifact 
buffer zones is accessed or re-accessed as necessary so that spatial locations included 
in the zones may be identified and filled according to special buffer zone filling 
operations in subsequent steps for determining/refining surface pixels in the 

20 composite image. Then, in step S964, the data corresponding to the current composite 
surface map is segmented into surface regions exhibiting limited focal plane range. 
That is, the current composite surface map is segmented into surface regions where 
the image data for all the spatial locations included in a surface region corresponds to 
a limited number of spatially adjacent source image focal planes. Operation then 

25 continues with step S965. 

[0128] It should be appreciated that, in various exemplary embodiments, if 
operations corresponding to step S963 are performed, then the data corresponding to 
the edge-artifact buffer zones are included in the current composite surface map. It 
should be further appreciated that, in various exemplary embodiments, spatially 

30 adjacent source image focal planes are conveniently indicated by sequential source 
image indexes. In one exemplary embodiment, the data corresponding to the current 
composite surface map is in the form of the previously described 5x5 surface meta- 
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pixels and generally includes a corresponding source image index which applies to 
each included pixel, hi this exemplary embodiment, a recursive region-growing 
algorithm is applied to the data corresponding to the current composite surface map. 
[0129] In one exemplary embodiment of this recursive region-growing 
5 technique, the current composite surface map data is systematically searched for the 
meta-pixel spatial location having the maximum surface-focus indicator value. For 
the reasons described below, the maximum surface-focus indicator value should also 
be greater that zero. Then, that meta-pixel spatial location is set as the primary seed 
for growing a surface region having a limited focal plane range. Next, the region is 

1 0 grown around the primary seed by including in the region all 4-connected neighbors of 
the primary seed which have a source image index which is within a range of plus or 
minus two increments of the source image index of the primary seed. Subsequently, 
each neighbor added to the region is taken as a secondary seed Each of the 4- 
connected neighbors of each secondary seed which have a source image index which 

15 is within a range of plus or minus two of the source image index of the primary seed 
are also included in the region. Each such added neighbor becomes an additional 
secondary seed. This is repeated until the region can grow no further. 

[0130] In various exemplary embodiments, if the total region size so 
determined is less than a threshold size, such as, for example, three meta-pixels, then 

20 the elements of that region are assigned a surface-focus indicator value of zero, and 
are returned to the pool of unassigned elements. The step of requiring a minimum 
threshold size tends to suppresses small surface artifacts due to noise and/or minute 
out-of-focus surface particles, and the like. Otherwise, any spatial locations which 
have been associated with a region in any current or previous region-growing iteration 

25 are not considered in subsequent recursions. 

[0131] When a region can grow no further, the recursive algorithm repeats 
for a new region, beginning with the search for the next spatial location having the 
maximum surface-focus indicator value. This next spatial location becomes a new 
primary seed for the new region. The recursive region-growing algorithm is repeated 

30 until all spatial locations included in the current composite surface map are included 
in the surface regions exhibiting limited focal plane range, or for a number of 
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iterations predetermined to leave only a "visually insignificant" number of scattered 
pixels which are not associated with a surface region of limited focal plane range. 

[0132] Throughout the recursive region growing algorithm, in various 
exemplary embodiments, the primary seeds are not generally selected corresponding 
5 to spatial locations in any edge-artifact buffer zone, so that out-of-focus edge artifacts 
can be further suppressed. In one exemplary embodiment, this is accomplished by 
assigning a special value of zero as the surface-focus indicator value for these edge- 
artifact buffer zone spatial locations, such that no such edge-artifact buffer zone 
spatial location will ever be a "maximum seed". 

10 [0133] However, it should be appreciated that it is generally desirable that 

spatial locations in edge-artifact buffer zones, as well as spatial locations in the 
regions that failed to attain the minimum size threshold described above, should be 
allowed to join adjacent surface regions that grow to include them. This is done 
simply because such spatial locations are most likely to be best-focused at the same 

15 focal plane as the immediately adjacent surface region. Thus, throughout the 

recursive surface region- growing algorithm, whenever a region grows to include a 
neighbor that has a surface-focus indicator value of zero, that neighbor is included in 
the region, regardless of the source image index of the current primary seed. 

[0134] It should be appreciated that, in the foregoing description of 

20 exemplary operations of the step S964, the range of source image indexes selected as 
the basis for admitting neighbors into a region will, in general, depend on the depth of 
focus of the plurality of source images relative to the focal plane step between 
adjacent source images. For example, in various respective exemplary embodiments, 
the nominal focal plane range allowed in a surface region may be set so that it does 

25 not extend beyond 1, 2, or 3 times the depth of field of the source image 

corresponding to the primary seed. In cases where the source image focal planes are 
relatively closely spaced, the number of source image indexes corresponding to the 
nominal focal plane range allowed in a region will be relatively greater. In contrast, in 
cases where the source image focal planes are relatively coarsely spaced, the number 

30 of source image indexes corresponding to the nominal focal plane range allowed in a 
region will be relatively fewer. It should also be appreciated that, in various 
alternative embodiments, 8 -connected neighbors may be considered instead of the 4- 



May 25, 2001 



MEIRef.: 2000-04 
O&BRef.: 106538 



42 

connected neighbors used in the previously described surface region-growing 
operations. 

[0135] It should be further appreciated that other surface region 
membership criteria may be added to overrule and/or supplement the simple source 
image index criteria described above, and that larger regions may thus result. For 
example, in a further exemplary embodiment, the distance of any particular neighbor 
from the primary seed is considered, and/or with the difference in the source image 
indexes between the neighbor and the primary seed, along with the associated surface- 
focus indicator values of the neighbor and the primary seed. The desired relationship 
between these various factors, which serves as the criteria for admitting a neighbor to 
a region, is determined by trial and error on a number of images representative of the 
expected application of the composite image system. In such an embodiment, slowly 
undulating or inclined surface portions which do not include a high degree of 
distinctive texture may be included in a region and a desirable composite image may 
still be obtained. 

[0136] It should also be appreciated that, in various exemplary 
embodiments, spatial voids may remain in the data corresponding to the composite 
surface map resulting from the foregoing surface-pixel determining operations. Thus, 
in such exemplary embodiments, operations are performed to identify such voids and 
assign the source image index of the nearest neighbor to the spatial locations 
associated with each void. If there is more than one nearest neighbor, then the source 
image index of any nearest neighbor may be assigned to the void, according to any 
technique which is convenient in the processing operations used to determine the 
nearest neighbor. 

[0137] In step S965, each of the surface regions determined in the 
operations of the previous step S984 are analyzed to determined the average source 
image index for each surface region. For each surface region, the average source 
image index for that region is the average of all the source image indexes of all the 
spatial locations included in that region. 

[0138] Next, in step S966, for each of the surface regions analyzed in steps 
S964-S965, all of the spatial locations included in a region are assigned the source 
image index which is closest in value to the average source image index for that 
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region. The average source image index corresponds to the source image having the 
focal plane which is taken to be most representative of all the respective focal planes 
corresponding to the respective spatial locations included in the region. Operation 
then continues to step S976, where operation continues to step S970. 

[0139] It should be appreciated that when the surface pixels of the 
composite image are constructed based on the source image indexes assigned in step 
S966, each portion of the composite image corresponding to a particular surface 
region will thus be based on a single source image. In this way, the foregoing 
exemplary operations suppress or eliminate at least some out-of-focus artifacts not 
otherwise eliminated by the systems and methods of this invention. 

[0140] As previously described, following execution of step S970 of Fig. 8, 
operation then continues to step SI 000 of Fig. 3. In one exemplary embodiment of 
the operations corresponding to step SI 000 of Fig. 3, a composite image 
representation resulting from the preceding steps of Fig. 9 may still contain voids. 
That is, a limited number of spatial locations may remain where the composite image 
pixels remain undetermined. For example, depending on the details of the particular 
surface pixel determining operations used, and especially in various exemplary 
embodiments which implement edge-artifact buffer zones, voids may be likely to 
occur at spatial locations near the edge-artifact buffer zones. In such a case, filling 
these and any other voids in the current composite image representation is preferably 
performed as part of step SI 000. 

[0141] In one exemplary embodiment of operations corresponding to step 
SI 000, the source images and/or source image indexes corresponding to all 
determined pixels in the composite image are known according to methods previously 
described. To fill the voids, the spatial locations in the composite image 
representation corresponding to each unique source image and/or source image index 
included in the composite image representation are operated on in a separate iteration 
of the following operation: Initially, one of the unique source image indexes is chosen 
as the source image index for the current iteration. Then the spatial locations in the 
composite image representation corresponding to that index are dilated using a 
structuring element of a prescribed size, for example 3x3 pixels, according to known 
methods of dilation. For each spatial location "added" by the dilation operations of 
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the current iteration, the current source image index is assigned as the basis for that 
spatial location in the composite image. However, it should be appreciated that, in 
this exemplary embodiment, a subsequent dilation operation never overrules the basis 
for a spatial location which has been previously determined in the composite image 
5 representation. Then these operations are repeated for a different source image index, 
until all source image indexes included in the composite image representation have 
been processed. If all voids are not filled after one pass through a set of iterations 
corresponding to all the source image indexes included in the representation of the 
composite image, the entire process is repeated. 

10 [0142] In an alternative exemplary embodiment of the preceding 

operations, a subsequent dilation operation may overrule a source image index basis 
for a spatial location which was previously determined, but only for spatial locations 
which were voids filled by a previous dilation operation. In one exemplary 
embodiment, if a particular spatial location has been assigned two or more source 

1 5 image indexes after a particular pass through all the included source image indexes, 
then the source image index corresponding to the highest valued surface-focus 
indicator, or edge-focus indicator, respectively, is used as the basis for that spatial 
location. Furthermore, if all corresponding indicators are not of the same type, then 
the source image index corresponding to the best surface-focus indicator is used. 

20 [0143] Returning now to a further discussion of operations corresponding 

to steps S600 and/or S700 of Fig. 3, in various exemplary embodiments of the systems 
and methods according to this invention, artifacts in the composite image are further 
suppressed and the composite image processing time is shortened when additional 
preliminary information is available regarding the portion of the object to be shown in 

25 the composite image. For example, information such as a CAD file representing an 
object, or a previous composite image of a substantially identical object, is frequently 
available in an industrial applications of machine vision systems. 

[0144] In the case of a CAD file representation, it should be appreciated 
that the locations of edges and boundaries in the CAD file representation may be 

30 determined manually, in a semi-automated fashion, or fully automatically from a CAD 
representation, by a variety of known methods of CAD file feature extraction. In such 
a case, the spatial locations of the corresponding edges and boundaries in a current set 
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of source images of a corresponding object may then be determined by a further 
variety of known manual, semi-automated, or automated methods of spatial 
congruence and/or feature congruence image processing. These methods may include, 
for example, coordinate matching, pattern matching, template matching, and the like. 
5 For example, such methods are routinely used for the inspection of the positions of 
edges and boundaries on objects in a variety of commercially-available machine 
vision systems, such as the Quick Vision series of vision inspection machines 
available from Mitutoyo America Corporation (MAC), located in Aurora, IL. 

[0145] When a desirable previous composite image according to the 

10 systems and methods of this invention is available, then the spatial locations of edge- 
pixels in that composite image are also available. Such a desirable previous 
composite image, of a substantially identical object, may be made spatially congruent 
with the current source images according to various well-known methods. These 
methods may include, for example, coordinate matching, pattern matching, template 

1 5 matching, image correlation, and the like. Thus, the corresponding spatial locations of 
edges in the current source images may be identified. In the case of such pre-existing 
CAD and/or desirable composite image representations, all operations described 
herein for determining edge-pixels in the composite image may be restricted to spatial 
locations in the vicinity of the determined edges in the pre-existing CAD or desirable 

20 composite image representations. Thus, artifacts and aberrations that might otherwise 
occur due to edge-analysis operations in other parts of the images are effectively 
suppressed. In addition, the related composite image processing considers fewer 
spatial locations and proceeds faster. 

[0146] Turning now to Figs. 10-13, Fig. 10 is a first source image having a 

25 focal plane located at a first plane of an electronic assembly object, exemplifying the 
source images usable to construct a composite image according to the systems and 
methods of this invention. It should be noted that the source image shown in Fig. 10 
is relatively well-focused at the right and left sides of the source image, but generally 
out-of-focus in the center portion. Fig. 1 1 is a second source image having a focal 

30 plane located at a second plane of the electronic assembly object, again exemplifying 
source images usable to construct the composite image according to the systems and 
methods of this invention. It should be noted that the source image shown in Fig. 1 1 
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is relatively well-focused at the central areas of the source image, but is generally out- 
of-focus at the left and right edge portions. The source images of Figs. 10 and 1 1 
were acquired using a commercially available vision system from the Quick Vision 
series of vision inspection machines available from Mitutoyo America Corporation 
5 (MAC), located in Aurora, IL. 

[0147] These two typical sources images are representatives from a 
plurality of eleven source images, wherein each pair of adjacent source images was 
separated by a 0. 1mm focal plane step. The gray scale information for the plurality of 
eleven source images was captured using a 2.5x magnification lens, the system 

10 camera, and operator-determined lighting settings. The source image shown in Fig. 
10 corresponds to the assigned source image index of 5, while the source image 
shown in Fig. 1 1 corresponds to the assigned source image index of 10. Thus, these 
two images correspond to focal planes which are separated by 0.5 mm. 

[0148] Fig. 12 is a representation of the composite image map determined 

15 based on the plurality of eleven source images, which were selected and analyzed 
according to the systems and methods of this invention. In Fig. 12, the data 
representation is shown in the literal form of a two-dimensional composite image 
map, for ease of visualization. Each shade of gray at a particular spatial location in 
Fig. 12 represents a different particular source image which is to be used as the basis 

20 for determining the composite image at that spatial location. 

[0149] Fig. 13 is the composite image determined according to the 
composite image map represented in Fig. 12. It should be noted that the composite 
image shown in Fig. 13 is relatively well-focused throughout the entire composite 
image. At each spatial location in the composite image shown in Fig. 13, the 

25 composite image is determined by simply copying the image value at that spatial 

location from the particular source image which is indicated by the composite image 
map as the basis for determining the image value of the composite image at that 
spatial location. No additional image processing has been used to further enhance the 
composite image of Fig. 13. However, additional enhancements usable to improve the 

30 subjective quality of the composite image are also within the scope of the systems and 
methods according to this invention. Such enhancements, include, but are not limited 
to, various filters and smoothing techniques known to those skilled in the art of image 
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processing. It should be appreciated that any appropriate known or later-developed 
image enhancement technique can be used to enhance the composite image. 

[0150] It should be appreciated that, the control system portion 120 shown in 
Fig. 1 is, in various exemplary embodiments, implemented using a programmed general 
purpose computer. However, the control system portion 120 can also be implemented 
on a special purpose computer, a programmed microprocessor or microcontroller and 
peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital 
signal processor, a hardwired electronic or logic circuit such as a discrete element 
circuit, a programmable logic device such as a PLD, PLA, FPGA or PAL, or the like. In 
general, any device, capable of implementing a finite state machine that is in turn 
capable of implementing the operations corresponding to the flowcharts shown in Figs. 
2-8, and as otherwise described herein, can be used to implement the control system 
portion 120. The particular form each of the circuits shown in the control system 
portion 120 of Fig. 1 will take is a design choice and will be obvious and predictable 
to those skilled in the art. 

[0151 ] In Fig. 1 , alterable portions of the memory 1 40 in various exemplary 
embodiments, whether volatile or non-volatile, can be implemented using any one or 
more of static or dynamic RAM, a floppy disk and disk drive, a writeable or rewriteable 
optical disk and disk drive, a hard drive, flash memory or the like. In Fig. 1 , the 
generally static portions of the memory 140 are, in various exemplary embodiments, 
implemented using ROM. However, the static portions can also be implemented using 
other non-volatile memory, such as PROM, EPROM, EEPROM, an optical ROM disk, 
such as a CD-ROM or DVD-ROM, and disk drive, flash memory or other alterable 
memory, as indicated above, or the like. Thus, in Fig. 1, the memory 140 can be 
implemented using any appropriate combination of alterable, volatile or non- volatile 
memory or non-alterable, or fixed, memory. 

[0152] Moreover, the control system portion 120 can be implemented as 
software executing on a programmed general purpose computer, a special purpose 
computer, a microprocessor or the like. In this case, the control system portion 120 
can be implemented as a routine embedded in the vision system 100, as a resource 
residing on a server, or the like. The control system portion 120 can also be 
implemented by physically incorporating it into a software and/or hardware system. 
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[0153] While this invention has been described in conjunction with the 
exemplary embodiments outlined above, it is evident that many alternatives, 
modifications and variations will be apparent to those skilled in the art. Accordingly, 
the exemplary embodiments of the invention, as set forth above, are intended to be 
illustrative, not limiting. Various changes may be made without departing from the 
spirit and scope of the invention. 
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